[WIP] Add msProbe response anomaly detection for service inference - #2

…ing (AISBench#472)

…h#486)

* docs: remove response anomaly design document from docs tree

* docs: complete response anomaly detection coverage in user docs

- Fix execution-order description: detection is serially bound to the
  inference stage, not parallel with Eval (cli_args/mode, zh+en)
- Add a dedicated detection_status table (completed/skipped/unavailable/
  failed) with troubleshooting hints
- Document result schema fields (anomaly_type_name, uuid) and clarify
  that anomaly results are an independent audit and never change
  accuracy/perf metrics
- Correct --reuse inheritance: matched by id+uuid, only completed cases
  are inherited; skipped/failed/unavailable are re-detected
- Document msprobe_config_path (threshold tuning entry) in model config
  examples and parameter tables
- Describe payload archive layout (zst shards + manifest), empty-manifest
  semantics, and stale build-dir cleanup
- Add detection log path and status file location for troubleshooting
- Add vLLM return_token_ids request parameters and version requirement
- Note model_name fallback to abbr with degradation warning

* docs: add on-disk layout tree for response anomaly outputs

Add a directory tree of <work_dir> covering all detection-related
artifacts with per-path explanations (zh+en):

- predictions: lightweight inference results without payload
- response_anomaly/<model>/<dataset>.jsonl: per-Case detection results
- payload_staging: transient during inference, cleaned after detection
- payload archive: zst shards + manifest, retention-mode semantics,
  empty-manifest directory meaning
- response_anomaly_config: auto-generated msProbe config layout
  (config.yaml thresholds, mtype_config.json, token2category map),
  non-overwrite and multi-model merge behavior
- logs/response_anomaly: detection log path

* docs: unify response_anomaly example comments for msprobe paths

Give msprobe_mtype_path and msprobe_token2category_dir the same
inline-comment style as msprobe_config_path (optional marker + purpose
description) in both zh/en cli_args.md and models.md, instead of bare
placeholder paths.

* docs: use double quotes for msprobe path placeholders in examples

* docs: align model_name fallback description with fail-fast behavior

Update the model_name resolution description in zh/en cli_args.md and
models.md to match the behavior change in the companion fix PR:

- model_name falls back to the model directory basename (de-facto model
  name, same default as the config generator), never to the model abbr
- when neither model_name nor a model directory is available (explicit
  msprobe paths only), startup fails fast with guidance instead of
  silently running detection with a wrong model name

* docs: rephrase model_name default as taking name from model path

Drop the negative 'abbr is never a fallback' phrasing (an artifact of
the old behavior) and state the default positively: the model name is
taken from the model path.

* docs: response anomaly switch is command-line only

Align docs with the behavior change in the companion fix PR:

- --response-anomaly is the only enable switch; omitting it disables
  detection (no --no-response-anomaly form)
- remove the config-file enabled=True examples; the config-file
  response_anomaly entry now documents only non-switch settings
  (payload_retention / payload_storage)
- drop the 'command line overrides the config file' precedence note

* docs: drop response_anomaly.enabled mentions from docs

The enabled key is not a supported config-file switch, but the code only
warns for the top-level block (model-level enabled keys are silently
unused), so do not advertise 'config enabled is ignored with a warning'
behavior. State only the positive CLI rule: --response-anomaly enables
detection, omitting it disables.

…ISBench#487)

* fix: derive response anomaly model_name from model path instead of abbr

When response_anomaly.model_name was not configured, ConfigManager fell
back to the model 'abbr'. The abbr is a task label (e.g.
'vllm-api-general-chat') unrelated to the served model, so msProbe
received a model name that would silently miss the keys in
mtype_config.json and token2category. In the explicit-msprobe-paths
setup this produced degraded detection results ( Cases still marked
'completed' while rare-character/garbled checks never matched); in the
auto-generation setup it keyed the generated configs by a meaningless
name that breaks later reuse with real model-name configs.

New behavior:
- model_name resolution order: model-level > global > basename of
  model 'path'/model_path (the de-facto model name, same default the
  config generator uses); the abbr is never used as a fallback
- When no reliable name can be derived (no model_name and no model
  directory), fail fast at config initialization with guidance,
  instead of silently proceeding with a wrong name
- The detector coordinator warning no longer claims an 'abbr fallback'
  that never actually happened (it passes None to the detector);
  it now truthfully states the detector runs without a model name

Tests: add basename-fallback and explicit-paths-require-model_name
cases; update explicit-paths and nonexistent-paths cases for the new
validation.

* feat: make --response-anomaly a command-line-only enable switch

The response anomaly enable switch was previously accepted from both
the CLI (--response-anomaly / --no-response-anomaly) and the config
file (response_anomaly.enabled), with the CLI taking precedence. This
made the single most important toggle of the feature a mixed-mode
setting and left a silent enable path in config files.

New behavior:
- --response-anomaly is the only way to enable detection
- --no-response-anomaly is removed: detection is simply disabled when
  the flag is absent
- response_anomaly.enabled in the config file is no longer supported:
  it is dropped with a warning pointing to the command-line switch, so
  a stale config value can never silently enable detection
- all non-switch response_anomaly settings (payload_retention,
  payload_storage, model-level msprobe resources) keep working as
  config-file options, and --response-anomaly-payload-retention still
  overrides the config from the CLI

The enabled value is now taken strictly from the parsed CLI boolean;
non-boolean values (missing attribute, mocks) resolve to disabled.

Tests: add cases for config-file enabled being ignored and for the
CLI switch as the sole enable path.

* fix: fall back model_name to global model_path before model path field

The global response_anomaly block (top-level, sibling of 'models') can
carry model_path, but the model_name fallback only looked at the
model-level model_path and then the model 'path' field. When a user
accidentally placed model_path in the top-level block, model_name ended
up being derived from the model 'path' field (wrong model name) or
raising when that field was empty, even though a valid model_path was
available in the global block.

Fix the fallback chain to prefer the most specific source:
  model-level model_name (explicit)
  -> model-level model_path basename
  -> global model_name
  -> global model_path basename
  -> model 'path' field basename

Tests: add cases for global-model_path fallback and model-level-beats-global priority.

* refactor: drop 'abbr is not a fallback' explanation from model_name error

Remove the 'The model abbr is a task label unrelated to the served
model and is not used as a fallback' sentence from the model_name
validation error. The negative framing is an artifact of the old
abbr-fallback behavior and adds no information for a user who just hit
the error; the remaining message states the requirement and how to fix
it, matching the positive wording already used in the docs.

* add mini dataset doc

* review fix

* multilingual mini fix

* add upload_pip.sh

* add testpypi only

* add testpypi only

* add testpypi only

* add testpypi only

* add testpypi only

* add testpypi only

* update install docs

* simple cmd for api

* add UT

* add UT

* add docs

---------

Co-authored-by: SJTUyh <yh_silence@alumni.sjtu.edu.cn>

* 数据集适配

* 提示词等完全对齐

* 修复已知环境变量问题

* 修复数据集加载问题

* 适配chat接口

* 优化裁判模型输入的提取逻辑：仅提取content部分，不带reasoning部分

* 补充资料

* 补充ut用例

* 资料补充 配套文件导入方式

* 初版代码

* fix prefix cache dataset and runtime edge cases

* 修复bug

* preserve requests artifact field order

* complete prefix cache detailed requirements

* 修复运行bug

* 添加readme解释

* 文档说明

* 打印日志

* 添加注释

* 修改前缀方法

* 生成前缀缓存数据集：仅保留 inspect/prepare/validate 离线功能

* 添加需求设计

* 添加修改

* 同步写日志

* 添加修改cli

* 删除文档

* 删除文档

* 删除文档

* 添加文档说明

* 门禁新增 prefix_cache 插件测试并补充用例提升覆盖率

- 门禁在 UT 之后执行 plugins/prefix_cache/tests，按 80/60 阈值校验覆盖率
- 新增 test_scenario/test_cli_flow/test_artifacts 用例，总覆盖率 77.92% -> 92.66%
- 修复 test_cli.py 中 execution_timestamp 传参位置错误

Co-Authored-By: Claude <noreply@anthropic.com>

* 冒烟测试增加 prefix_cache 变更触发条件

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

The parenthetical noting the absence of a --no-response-anomaly flag adds noise rather than information: omitting --response-anomaly already implies detection is disabled.

… tutorial

The response anomaly detection section was previously embedded in cli_args.md, where it split the Configuration Constant File Parameters heading from its content (zh) and mixed feature-level guidance into a parameter reference page. Move it into a standalone advanced tutorial page (zh/en) restructured as a complete guide from dependency installation to configuration, fix the orphaned heading, point the --response-anomaly CLI rows to the new page, update all cross links (models/mode/accuracy_benchmark), and register the new page in the toctree.

The wording 不增加即关闭/不增加则关闭 (omitting it disables detection) reads like omitting the flag actively disables the feature; reword to 不增加默认不开启 (not enabled by default) and align the English wording accordingly.

The feature needs no config-file fields in practice: the model path field (always set for real runs) drives model-name derivation and msProbe config auto-generation, and payload retention is controlled by the --response-anomaly-payload-retention CLI switch. Drop the global and model-level response_anomaly config-block guidance from the tutorial, the models.md code sample and parameter table, and rework Quick Start around the two CLI switches.

User-facing docs should not expose the internal detector implementation name. Replace msProbe/mindstudio-probe wording with neutral terms (built-in anomaly detection, detector source, detection configs) across the tutorial, cli_args/mode/accuracy_benchmark notes and the install guide, zh and en. The pip extra name response_anomaly remains the single user-facing handle for the dependency.

…el repositories

The path directory basename is the model-name source for detection; document that it must stay consistent with the model name on Hugging Face / ModelScope / Modelers and must not be renamed arbitrarily, in the models.md path row and the anomaly detection tutorial tip (zh/en).

The bare value/behavior table header did not identify which parameter the values belong to; label it with --response-anomaly-payload-retention (zh/en).

The detection flow step describes payload staging only; the prediction-file behavior belongs to the main inference flow and is out of scope here (zh/en).

Retention and staging cleanup both happen during archive finalization; use and/及 instead of or (zh/en).

Detection cleanup only removed the per-dataset staging subtree (or renamed it away in all mode), leaving an empty payload_staging directory under response_anomaly/<model>. rmdir the staging root after the per-dataset cleanup; it only succeeds when the last dataset of the model finishes detection, so multi-dataset runs stay safe. Covered by a new UT; wording in the zh/en tutorial updated accordingly.

The shard checksum value was formatted as sha256:<hex> while the field is already named sha256, duplicating the algorithm name. Store the bare hex digest in both manifest builders and strengthen the UT to compare against the hash recomputed from the shard file.

…ion log

The finalize step logged the aggregated summary with the exact same wording as the per-task completion line, making them indistinguishable in the console. The summary line now reads Response anomaly detection summary across N task(s). Per-task completion logs and dedicated task log files are unchanged; a new UT covers multi-dataset aggregation (summary totals) and per-task log isolation.